# imports
from google.colab import drive
from IPython.display import Image
import matplotlib.pyplot as plt
import matplotlib
import numpy as np
import pandas as pd
import seaborn as sns
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
Introduction¶
In this challenge, our goal was to create a classifier for a Brain-Computer Interface (BCI). A BCI is a device that intends to map mental states to computer commands. In our case it was an Eye-Brain-Computer-Interface, thus the main objective of this paradigm is to replace the computer mouse with electroencephalograms. We usually look at an object when we are going to manipulate it. Thus, eye tracking can be used to communicate intended actions. An effective human-machine interface, however, should be able to differentiate intentional and spontaneous eye movements. This is actually close to mentalism. We will track the EEG results through a little experiment.
Our work is divided into three different parts:
- Data exploration
- Model Selection
- Performance evaluation
Context¶
During the experiment, a BCI user has to play a simple game - align balls of one color in lines. A transaction to collect individual data looks as follows. First, the user investigates the playing field and makes a decision about which ball to move. During this part of game, the EEG fragments corresponding to eye focus are collected and labeled as Non-Target (because the user has just been observing the playing field). Then, when a decision is made, the user focuses on the “Control mode ON” button. In this mode, each eye focus of 500 ms length is treated as a mouse click. Then the user focuses on the ball to move (this ball becomes highlighted). Then the user focuses on the free cell of the board to place this ball. These focusing actions are intentional, and the corresponding EEG fragments are labeled as Target.
1. Data exploration¶
We report an electroencephalogram (EEG) marker that differentiates gaze fixations used for control from spontaneous fixations involved in visual exploration. So there is two classes the Target ones (y=1) and the Non-Target (y=0).
We are expecting a slow negative wave with a maximum in the parietooccipital region was present in each participant's averaged FRPs in the control-on conditions and was absent or had much lower amplitude in the control-off condition. This wave was similar but not identical to stimulus-preceding negativity, a slow negative wave that can be observed during feedback expectation.
Concerning the data, we have 13 individuals and for each ot them we have an array 'X' like this: EEG epochs (Trials) x Time x Channels and an array 'y' that stores the labels 1 or 0 that correspond to the epoch.
1.1 Study on one individual¶
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Mean_amplitude.png')
As we can see on the chart right above, there is always a peak at the beginning of class Target and then it decreases. This is probably where we can interpret the signal as Target label. However it seems to be more random when the signal is labeled as Non-Target so when the person is having a look a the game without making any decisions.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Distribution_peak_index.png', width=1000, height=900)
The remarks we had previously appears to be quite wrong according to the fact in average over all the channels there is no peak at the beginning of a Target sequence. Actually it could be linked to the fact that in average the brain is implicated every time. While in the Non-Target sequence we can observe a peak a the beginning in average which could be related to the first glance on the game and all the understanding one needs to do when one first faces a game. Then the amplitude turns to decrease because the patient understands what he has to do so he doesn't think that much.
Now we will try to understand where are the most solicited part of the brain according to the EEG results.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/EEG_T_NT.png', width=1000, height=480)
As we could have expected, we see a slow negative wave with a maximum in the parietooccipital region was present in each participant's averaged FRPs in the control-on conditions and was absent or had much lower amplitude in the control-off condition. So we decided to plot the distribution of the most active part of the brain, which is PO8.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Distribution_peak_index_PO8.png', width=1000, height=900)
1.2 Study over all individuals¶
1.2.1 Temporal Information¶
In the previous exploration part, we had a closer look on how the Non-Target and Target can be classified and moreover we have perceived that PO8 is very implicated in this difference. Thus, we will try to see if it is the same for all individuals.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Mean_amplitude_all_indiv_PO8_T.png', width=1000, height=500)
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Mean_amplitude_all_indiv_PO8_NT.png', width=1000, height=500)
As we can see up above, there is a similarity between all these individuals. We can see that along PO8 they all have the same shape in amplitude for Non-Target sequence. They mainly decrease through the time. While for the Target point of view we can see that it is as fuzzy than before for everyone as we had seen before when we studied only one individual.
To sum up, we can guess that one way to classify the sequence could be to have a look for a global decreasing. If there is one then it must be a Non-Target sequence, if there is none it must be a Target one.
1.2.2 Spatial Information¶
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/EEG_T_NT_indiv_35.png', width=1000, height=400)
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/EEG_T_NT_indiv_250.png', width=1000, height=400)
Finally we see up above that when it is the beginning of a Non-Target sequence it is the parieto occipital part of the brain that is solicited whereas when it is the end of the sequence it the frontal part of the brain. In other words, the parieto occipital part is well the one that is used for the individual to be focus. This part is solicited uniformly when it is a Target sequence while it is decreasingly solicited when it is a Non-Target sequence. Indeed when you have understood the rules, you don't really need to focus anymore.
1.2.3 Spectral Information¶
We tried to see if there was a difference between Non-Target and Target sequence by computing the average in time of the power measure after having applied a band pass filter as suggested in the following paper: https://www.researchgate.net/publication/281076368_A_Tutorial_on_EEG_Signal_Processing_Techniques_for_Mental_State_Recognition_in_Brain-Computer_Interfaces
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Band_pass_power.png', width=1000, height=300)
However we didn't really observed any distinctive patterns between Non-Target and Target Sequence as we could have seen when we looked at the Temporal and Spatial Information.
2. Model Selection¶
2.1. EEGNetv4¶
2.1.a. Model description¶
This baseline model is described in details in the paper EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces, from https://arxiv.org/abs/1611.08024
In the explanatory notebook are presented the basic building blocks to construct an EEG statistical model. It can be summed up:
Spatial filters -> Temporal filters -> Classifier
Spatial filter: This filter aims to construct a source representation of information, observed on sensors. A usual algorithm used is Common Spatial Pattern. It relies on the assumptions that:
- frequency band and time window are known
- band-passed signal is jointly gaussian within the time window
- source activity constellation differs between two classes
To compute it:
- treat it as an Optimization Problem
- treat it as an Eigenvalue solving problem
- treat it geometrically
Temporal filter: It is used because different sets of sources produce signals within different frequency bands.
Classifier: It can be a simple linear classifier such as LDA, SVM...
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/eegnetv4.PNG', width=800, height=400)
Let us explain briefly the structure of this condesned Convolutional Neural Network.
The first layer (Conv2D) is a temporal layer, and it splits the data into different frequency bands.
The second layer is a Separable Convolution (DepthwiseConv2D). It aims to reduce the number of parameters to fit and also to decouple the relationship within and across feature maps by first learning a kernel summarizing each feature map individually, then optimally merging the outputs afterwards. To sum-up, each filter is being adapted to the frequency band of interest.
The third layer is another temporal layer followed by a spatial frequency filter (SeparableConv2D).
Then, the classificaiton layer uses a softmax classification.
The two parameters that we have to tune are:
- F1 which fits the number of feature maps containing the EEG signal at different band-pass frequencies (Conv2D layer).
- D a depth parameter controlling the number of spatial filters to learn for each feature map
2.1.b. Model tuning¶
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/overfitting.PNG', width=800, height=400)
The graphs above are taken from the execution of the baseline model with a number of training epochs of 200 (max). They show the evolution of the train and validation loss (in blue and in orange respectively) with respect to the training epoch. We can see that the model overfits. We decided then to reduce the number of training epochs to 25, to avoid overfitting and to have less computation time. Indeed, to run the baseline model with 200 training epochs on all patients and using a GPU accelerator takes over 20 minutes. With 25 training epochs it takes about 3 minutes allowing us to do some hyperparameter tuning more easily.
Hyperparameter tuning
We tried to tune the two parameters F1 and D. To do this we ran a GridSearch around the original values giving us many sets of parameters. Here, we report some of our results.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/selecting.PNG', width=800, height=400)
The histograms above show the AUC validation loss for different sets of parameters. We can see that the optimal set of parameters is not always the same. It could have been interesting to try to hypertune for each patient separately. This is one of the main problem for BCI issues: generalizing to all individuals. Each one of us has a unique brain, behaviour etc.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/selecting_mean.PNG', width=400, height=300)
In order to choose the best parameters, we took the mean over all patients. This gave us values of D=3 and F1=11
2.2. Feature extraction | Band-pass Filter, CSP¶
First we tried some feature extraction using the mean power for most relevant part of the brain. We decided to work on the beta bandwidth [13-30]Hz, plus we decided to focus on the difference between the parieto occipital part and the frontal part so we focused on the two followings Fz and POz. We obtained linearly non separable data.
Image(filename='/content/drive/My Drive/AML/Chal3 - EEG Classification/Images/Scatter.png', width=400, height=300)
Thus, according to the exploration we did up above, we realized that it could be relevant to do spatial filetring using the CSP algorithm, Common Space Pattern, to identify which part is actually the most representative for each of the Target or Non-Target sequence.
However, we had real difficulties to implement the CSP algorithm correctly in order to extract feature and deal with the problem as a classifier problem because of the badness of the feature extraction. Too many parameters were involved, the good bandwith was tough to find and which area was the most important to differentiate these two classes was hard find also.
3. Performance evaluation¶
patients = [25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 38]
auc_test = [0.58, 0.67, 0.72, 0.80, 0.74, 0.63, 0.52, 0.78, 0.78, 0.70, 0.59, 0.65, 0.63]
df = pd.DataFrame(np.transpose([patients, auc_test]), columns=['Subject','Test AUC'])
df
Here, the testing set is chosen randomly from the whole dataset, for each subject. We finally obtained interesting results. Some test AUC are quite close to 1 (0.80 for subject number 28) and for others it is not as good (0.52 for subject number 32). On average, we obtain a Test AUC of 0.68, and it took 150 seconds to execute.
It is very difficult to choose a model that would predict well for all subjects, each one of us behaves differently. With a more global understanding of the subject and the data it could be easier to:
- implement the algorithms that could be used for regularization (adversarial examples)
- modify the structure of the neural network using additional spatial or temporal filters: this would require a very good understanding of the data. We tried to do this in the data exploration part, but it was difficult to implement as filters (layers of a neural network)
- using more adequate classifiers (LDA Classifier for instance)
Conclusion¶
While we did quite well on hypertuning the EEGNetv4 for some of the subjects, such as number 28, it is still hard to build a classifier that is robust to all type of subjects. Indeed we tried to do feature extraction to focus mainly on relevant features but we had real difficulties in extracting these relevant informations from complex raw EEG.
So we think that one main next workstream should be to find a feature extraction method that fits to this classification task. In order to build a robust classifier that classify correctly these two classes and deal with the issue as a linear classifier or simply use a Logistic Regression for instance.